Efficient Large-Scale Visual Representation Learning

In this article, we present our approach to single-modality visual representation learning. Understanding visual representations of product content is vital for recommendations, search, and advertising applications in e-commerce. We detail and contrast techniques used to fine-tune large-scale visual representation learning models in an efficient manner under low-resource settings, including several pretrained backbone architectures, both in the convolutional neural network as well as the vision transformer family. We highlight the challenges for e-commerce applications at-scale and highlight the efforts to more efficiently train, evaluate, and serve visual representations. We present ablation studies evaluating the representation offline performance for several downstream tasks, including our visually similar ad recommendations. To this end, we present a novel text-to-image generative offline evaluation method for visually similar recommendation systems. Finally, we include online results from deployed machine learning systems in production at Etsy

adSformers: Personalization from Short-Term Sequences and Diversity of Representations in Etsy Ads

In this article, we present a general approach to personalizing ads through encoding and learning from variable-length sequences of recent user actions and diverse representations. To this end we introduce a three-component module called the adSformer diversifiable personalization module (ADPM) that learns a dynamic user representation. We illustrate the module's effectiveness and flexibility by personalizing the Click-Through Rate (CTR) and Post-Click Conversion Rate (PCCVR) models used in sponsored search. The first component of the ADPM, the adSformer encoder, includes a novel adSformer block which learns the most salient sequence signals. ADPM's second component enriches the learned signal through visual, multimodal, and other pretrained representations. Lastly, the third ADPM "learned on the fly" component further diversifies the signal encoded in the dynamic user representation. The ADPM-personalized CTR and PCCVR models, henceforth referred to as adSformer CTR and adSformer PCCVR, outperform the CTR and PCCVR production baselines by $+2.66\%$ and $+2.42\%$, respectively, in offline Area Under the Receiver Operating Characteristic Curve (ROC-AUC). Following the robust online gains in A/B tests, Etsy Ads deployed the ADPM-personalized sponsored search system to $100\%$ of traffic as of February 2023